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CLAIMS 



[Claim(s)] 

[Claim 1]An image generation system comprising: 

An imaging means which picturizes real space in the sight line direction in an observer's view 
position. 

A detection means to detect said observers view position and a sight line direction. 

A virtual space video generating means which generates a virtual space image in an observer's 

view position and a sight line direction which were detected by said detection means. 

A compound video generating means which generates a compound image which compounded a 

virtual space image generated by said virtual space video generating means and a real space 

image outputted from said imaging means, A management tool which manages in common 

information on an object which exists in each and its position, and a direction to a displaying 

means which displays a compound image generated by said compound video generating means, 

and real space and virtual space. 

[Claim 2]The image generation system according to claim 1, wherein said management tool can 
update each object of said real space and said virtual space, its position, a direction, and 
information on a state. 

[Claim 3]The image generation system according to claim 2, wherein said management tool 
notifies each object of said real space and said virtual space, its position, a direction, and 
information on a state to said compound video generating means for every predetermined time. 
[Claim 4]The image generation system according to claim 3, wherein said virtual video generating 
means presupposes that it responds to an information update from said management tool and 
generates a virtual space image based on each object of said real space and said virtual space, 
its position, and a direction. 

[Claim 5]The image generation system according to claim 3 or 4, wherein said compound video 
generating means starts drawing of said real space image according to an information update 
from said management tool. 

[Claim 6]The image generation system according to claim 5 after it ends drawing of said real 
space image and said compound video generating means is generated [ a virtual space image ] by 
said virtual space video generating means, wherein it performs composition with said real space 
image and said virtual space image. 

[Claim 7]The image generation system according to claim 6 just before said compound video 
generating means's performing composition of said real space image and said virtual space image, 
wherein it performs regeneration of said virtual space image based on an observer's view 
position and a sight line direction which were detected by said detection means. 
[Claim 8]According to an information update from said management tool, said virtual space video 
generating means, Perform processing which generates a virtual space image based on each 
object of said real space and said virtual space, its position, and a direction, and said compound 
video generating means, The image generation system according to claim 3 arranging in parallel 
processing which starts drawing of said real space image with generation processing of said 
virtual space image by said virtual space video generating means, and performing it according to 
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an information notification from said management tool. 

[Claim 9]An image generation system of any one statement of claim 1 thru/or 8, wherein the 
number of said observers is plurality, 

[Ciaim 10]An image generation system of any one statement of claim 1 thru/or 9 having a 

motion detecting means which detects operation including an observers gesture made applicable 

to detection, a state, etc. based on a detection result of said detection means. 

[Claim 11]The image generation system according to claim 10, wherein it is possible to use an 

observer's operation detected by said motion detecting means as an influence input to an object 

which exists in said compound real space and this compound real space, 

[Claim 12] An image generation method comprising: 

A process of being the image generation method for compounding a virtual space image on a real 

space image in the sight line direction in an observers view position, and generating a compound 

image, and detecting said observers view position and a line of sight position. 

A process of acquiring a real space image in said observers view position and a sight line 

direction, 

A process of obtaining management information including an object which exists in each of real 
space and virtual space and its position, and a direction. 

A process of generating a virtual space image in said observers view position and a sight line 
direction according to said management information, and a process of generating a compound 
image which compounded said virtual space image and said real space image according to said 
management information. 

[Claim 13]An image generation method according to claim 12 having the process of updating said 
management information. 

[Claim 14] An image generation method according to claim 12 or 13, wherein said management 
information is notified to a process of generating said compound image for every predetermined 
time. 

[Claim 15]An image generation method according to claim 14, wherein generation of said virtual 
space image is generated based on each object of said real space and said virtual space, its 
position, and a direction according to renewal of said management information. 
[Claim 16]An image generation method according to claim 14 or 15 characterized by starting 
drawing of said acquired real space image ignited by renewal of said management information at 
a generate time of said compound image. 

[Claim 17] An image generation method according to claim 16 after drawing of said real space 
image is completed and a virtual space image is generated generation of said compound image by 
said virtual space image generation process, wherein it is performed by compounding said real 
space image and said virtual space image. 

[Claim 18]An image generation method according to claim 17 just before performing composition 
of said real space image and said virtual space image, wherein regeneration of said virtual space 
image is performed based on a view position and a sight line direction of said detected observer. 
[Claim 1 9]According to renewal of said management information, generation of a virtual space 
image based on said management information is started, An image generation method according 
to claim 14 which starting drawing of said acquired real space image accompanying generation of 
said compound image, and carrying out by generation of said virtual space image and drawing of 
said real space image standing in a row. 

[Claim 20]An image generation method of any one statement of ciaim 12 thru/or 19, wherein the 
number of said observers is plurality. 

[Claim 21]An image generation method according to claim 20 having the process of detecting 
operation including an observer's gesture made applicable to detection based on said observer's 
view position and a line of sight position, a state, etc. 

[Claim 22]An image generation method according to claim 21, wherein it is possible to use said 
an observer's detected operation as an influence input to an object which exists in space of said 
compound image and space of this compound image. 

[Claim 23]An imaging means which picturizes a real space image in the sight line direction in an 
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observer s view position characterized by comprising the following, An image generation system 
provided with a detection means to detect said observers view position and a sight line 
direction, and a displaying means for displaying a compound image which compounded a virtual 
space image in said observer's view position and a sight line direction to said real space image. 
A detecting module which is the storage which stored a program for generating said compound 
image, and in which computer reading is possible, and detects an observer's view position and a 
sight line direction where said program was detected by said detection means. 
A virtual space image generating module for generating a virtual space image from an observer's 
view position and a sight line direction detected with said detecting module. 
A compound image generating module for generating said compound image from a virtual space 
image generated with said virtual space image generating module, and said real space image. 
An administrative module for managing an object which exists in each of real space and virtual 
space and its position, and a direction. 

[Claim 24]The storage according to claim 23, wherein said administrative module can perform 
renewal of each object of said real space and said virtual space, its position, a direction, and a 
state. 

[Claim 25]The storage according to claim 24, wherein said administrative module notifies each 
object of said real space and said virtual space, its position, a direction, and a state to said 
compound video generating means for every predetermined time. 

[Claim 26]The storage according to claim 25, wherein said virtual space image generating module 
generates a virtual space image based on each object of said reai space and said virtual space, 
its position, and a direction ignited by an information update from said administrative module. 
[Claim 27]The storage according to claim 25 or 26, wherein said compound image generating 
module starts drawing of said real space image ignited by an information update from said 
administrative module. 

[Claim 28]The storage according to claim 27 after it ends drawing of said real space image and 
said compound image generating module is generated [ a virtual space image ] by said virtual 
space image generating module, wherein it performs composition with said real space image and 
said virtual space image. 

[Claim 29]The storage according to claim 28 just before said compound image generating 
module s performing composition of said real space image and said virtual space image, wherein 
it performs regeneration of said virtual space image based on an observers view position and a 
sight line direction which were detected with said detecting module. 

[Claim 30]Said virtual space image generating module carries out an opportunity [ an information 
update from said administrative module ], Perform processing which generates a virtual space 
image based on each object of said real space and said virtual space, its position, and a direction, 
and said compound image generating module, The storage according to claim 25 arranging in 
parallel processing which starts drawing of said real space image with generation processing of 
said virtual space image by said virtual space image generating module, and performing it 
according to an information notification from said administrative module. 

[Claim 31]A storage of any one statement of claim 23 thru/or 30, wherein the number of said 
observers is plurality. 

[Claim 32] A storage of any one statement of claim 23 thru/or 31, wherein said program has a 
detecting module of operation which detects operation including an observer's gesture made 
applicable to detection based on a detection result of said detecting module, a state, etc. 
[Claim 33]The storage according to claim 32, wherein it is possible to use an observer's 
operation detected with said detecting module of operation as influence on an object which 
exists in said compound image space and this compound image space. 
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DETAILED DESCRIPTION 



[Detailed Description of the Invention] 
[0001] 

[Field of the Invention]This invention compounds the real space image and virtual space images, 
such as computer graphics, which were incorporated from the photographing device of a video 
camera etc., and relates to the image generation system, the image generation method, and 
storage which generate a compound image. 
[0002] 

[Description of the Prior Art]As a mixed reality device used as a display, the conventional HMD 
(head mount display), There are a proposal, now a thing which is by Oshima, Sato, Yamamoto, 
Tarnura, etc. (Oshima, Sato, Yamamoto, and Tamura: references, such as "realization of an AR2 
hockey:cooperation type mixed reality system", a Virtual Reality Society of Japan paper 
magazine, Vol.3, No.2, pp.55-60, and 1998.). 
[0003] 

[Problem(s) to be Solved by the Invention] However, in the above-mentioned conventional device, 
if an observer shakes a head at right and left, a real space image will follow in footsteps 
immediately, but phenomena, like a virtual space image follows later than a real space image are 
checked. That is, a time gap of a real space image and a virtual space image is conspicuous. 
[0004]The purpose of this invention reduces the time lag of a real space image and a virtual 
space image, and there is in providing the image generation system, the image generation 
method, and storage which can provide an observer with the compound image which has a 
feeling of devotion more. 
[0005] 

[Means for Solving the Problem]An imaging means in which the invention according to claim 1 
picturizes real space in the sight line direction in an observer s view position, A detection means 
to detect said observer's view position and a sight line direction, and a virtual space video 
generating means which generates a virtual space image in an observers view position and a 
sight line direction which were detected by said detection means, A compound video generating 
means which generates a compound image which compounded a virtual space image generated 
by said virtual space video generating means and a real space image outputted from said imaging 
means, It has a management tool which manages in common information on an object which 
exists in each and its position, and a direction to a displaying means which displays a compound 
image generated by said compound video generating means, and real space and virtual space, 
[0006]The invention according to claim 2 is characterized by the ability of said management tool 
to update each object of said real space and said virtual space, its position, a direction, and 
information on a state in the image generation system according to claim 1. 

[0007]In the invention according to claim 3, in the image generation system according to claim 2, 
said management tool notifies each object of said real space and said virtual space, its position, 
a direction, and information on a state to said compound video generating means for every 
predetermined time. 

[0008]In the image generation system according to claim 3, said virtual video generating means 
presupposes that the invention according to claim 4 is responded to an information update from 
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said management tool, and it generates a virtual space image based on each object of said real 
space and said virtual space, its position, and a direction. 

[0009]ln the invention according to claim 5, in the image generation system according to claim 3 
or 4, said compound video generating means starts drawing of said real space image according to 
an information update from said management tool. 

[0010]In the image generation system according to claim 5 f as for the invention according to 
claim 6, said compound video generating means performs composition with said real space image 
and said virtual space image, after it ends drawing of said real space image and a virtual space 
image is generated by said virtual space video generating means. 

[001 1]In the image generation system according to claim 6 T the invention according to claim 7 
said compound video generating means, Just before performing composition of said real space 
image and said virtual space image, regeneration of said virtual space image is performed based 
on an observer s view position and a sight line direction which were detected by said detection 
means. 

[0012]In the image generation system according to claim 3, the invention according to claim 8 
said virtual space video generating means, According to an information update from said 
management tool, each object of said real space and said virtual space, Perform processing 
which generates a virtual space image based on the position and direction, and said compound 
video generating means, According to an information notification from said management tool, 
processing which starts drawing of said real space image is arranged in parallel with generation 
processing of said virtual space image by said virtual space video generating means, and is 
performed. 

[0013]The invention according to claim 9 is characterized by the number of said observers being 
plurality in an image generation system of any one statement of claim 1 thru/or 8. 
[0014]The invention according to claim 10 has a motion detecting means which detects 
operation including an observer's gesture made applicable to detection, a state, etc. based on a 
detection result of said detection means in an image generation system of any one statement of 
claim 1 thru/or 9, 

[0015]The invention according to claim 1 1 is characterized by it being possible to use an 
observer's operation detected by said motion detecting means as an influence input to an object 
which exists in said compound real space and this compound real space in the image generation 
system according to claim 1 0. 

[001 6]A process of the invention according to claim 12 being the image generation method for 
compounding a virtual space image on a real space image in the sight line direction in an 
observers view position, and generating a compound image, and detecting said observer's view 
position and a line of sight position, A process of acquiring a real space image in said observer's 
view position and a sight line direction, A process of obtaining management information including 
an object which exists in each of real space and virtual space and its position, and a direction, 
According to said management information, it has a process of generating a virtual space image 
in said observer s view position and a sight line direction, and the process of generating a 
compound image which compounded said virtual space image and said real space image 
according to said management information, 

[0017]The invention according to claim 13 has the process of updating said management 
information, in an image generation method according to claim 12. 

[0018]As for the invention according to claim 14, in an image generation method according to 
claim 12 or 13, said management information is notified to a process of generating said 
compound image for every predetermined time. 

[0019]As for the invention according to claim 15, in an image generation method according to 
claim 14, generation of said virtual space image is generated based on each object of said real 
space and said virtual space, its position, and a direction according to renewal of said 
management information. 

[0020]As for the invention according to claim 1 6, in an image generation method according to 
claim 14 or 15, drawing of said acquired real space image is started by generate time of said 
compound image ignited by renewal of said management information. 
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[0021]In an image generation method according to claim 16 t the invention according to ciaim 17 
generation of said compound image, After drawing of said real space image is completed and a 
virtual space image is generated by said virtual space image generation process, it is carried out 
by compounding said real space image and said virtual space image. 

[0022] Just before the invention according to claim 18 performs composition of said real space 
image and said virtual space image in an image generation method according to claim 17, 
regeneration of said virtual space image is performed based on a view position and a sight line 
direction of said detected observer. 

[0023]In an image generation method according to claim 14 the invention according to claim 19, 
According to renewal of said management information, generation of a virtual space image based 
on said management information is started, drawing of said acquired real space image 
accompanying generation of said compound image is started, and it is carried out by generation 
of said virtual space image and drawing of said real space image standing in a row. 
[0024]The invention according to claim 20 is characterized by the number of said observers 
being plurality in an image generation method of any one statement of claim 12 thru/or 19. 
[0025]The invention according to claim 21 has the process of detecting operation including an 
observer's gesture made applicable to detection based on said observer's view position and a 
line of sight position, a state, etc., in an image generation method according to claim 20, 
[0026]The invention according to claim 22 is characterized by it being possible to use said an 
observer's detected operation as an influence input to an object which exists in space of said 
compound image and space of this compound image in an image generation method according to 
claim 21. 

[0027]An imaging means which picturizes a real space image [ in / this invention / in an 
observer's view position / in the invention according to claim 23 / the sight line direction ] r An 
image generation system provided with a detection means to detect said observer's view 
position and a sight line direction, and a displaying means for displaying a compound image which 
compounded a virtual space image in said observers view position and a sight line direction to 
said real space image is characterized by comprising: 

A detecting module which is the storage which stored a program for generating said compound 

image, and in which computer reading is possible, and detects an observer's view position and a 

sight line direction where said program was detected by said detection means. 

A virtual space image generating module for generating a virtual space image from an observers 

view position and a sight line direction detected with said detecting module. 

A compound image generating module for generating said compound image from a virtual space 

image generated with said virtual space image generating module, and said real space image. 

An administrative module for managing an object which exists in each of real space and virtual 

space and its position, and a direction. 

[0028]The invention according to claim 24 is characterized by the ability of said administrative 
module to perform renewal of each object of said real space and said virtual space, its position, a 
direction, and a state in the storage according to claim 23. 

[0029]In the invention according to claim 25, in the storage according to claim 24, said 
administrative module notifies each object of said real space and said virtual space, its position, 
a direction, and a state to said compound video generating means for every predetermined time. 
[0030]In the storage according to claim 25, the invention according to claim 26 said virtual space 
image generating module, A virtual space image based on each object of said real space and said 
virtual space, its position, and a direction is generated ignited by an information update from said 
administrative module. 

[0031]In the invention according to claim 27, in the storage according to claim 25 or 26, said 
compound image generating module starts drawing of said real space image ignited by an 
information update from said administrative module. 

[0032]In the storage according to claim 27, the invention according to claim 28 said compound 
image generating module, After it ends drawing of said real space image and a virtual space 
image is generated by said virtual space image generating module, composition with said real 
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space image and said virtual space image is performed. 

[0033]In the storage according to claim 28, the invention according to claim 29 said compound 
image generating module. Just before performing composition of said real space image and said 
virtual space image, regeneration of said virtual space image is performed based on an observer's 
view position and a sight line direction which were detected with said detecting module. 
[0034]In the storage according to claim 25, the invention according to claim 30 said virtual space 
image generating module, Ignited by an information update from said administrative module, each 
object of said real space and said virtual space, Perform processing which generates a virtual 
space image based on the position and direction, and said compound image generating module, 
According to an information notification from said administrative module, processing which starts 
drawing of said real space image is arranged in parallel with generation processing of said virtual 
space image by said virtual space image generating module, and is performed. 
[0035]The invention according to claim 31 is characterized by the number of said observers 
being plurality in a storage of any one statement of claim 23 thru/or 30. 

[0036]The invention according to claim 32 has a detecting module of operation which detects 
operation including a gesture of an observer who makes said program applicable to detection 
based on a detection result of said detecting module, a state, etc. in a storage of any one 
statement of claim 23 thru/or 31. 

[0037]The invention according to claim 33 is characterized by it being possible to use an 
observer's operation detected with said detecting module of operation as influence on an object 
which exists in said compound image space and this compound image space in the storage 
according to claim 32. 
[0038] 

[Embodiment of the Invention] Hereafter, it explains, referring to drawings for an embodiment of 
the invention. 

[0039]The appearance lineblock diagram of the image generation system which displays the 
compound image which dr aw ing 1 requires for one gestalt of operation of this invention, and 
drawing 2 are the perspective views showing the composition of HMD with which the head of the 
observer of drawing 1 is equipped. 

[0040]In this embodiment, as shown in drawin g 1 , a system shall be installed in the interior of a 
room about 5 m around, and the three observers 100a, 100b t and 100c shall experience mixed 
reality by this system. The setting position of this system, a size, and the number of observers 
are not limited to this, and can be changed freely. 

[0041]Each observers 100a f 100b, and 100c A compound real space image, HMD(head mount 
dispiay)101 for providing each observer with (it is hereafter called MR space image) on a head. A 
right hand (for example, dominant hand) is equipped with the hand part position direction sensor 
receiver 103 for detecting the position and direction of a hand for the head position direction 
sensor receiver 102 which detects the position and direction of a head on a head, respectively. 
[0042]This HMD101 is provided with the following. 

As show n in drawing 2 (a) and (b), they are the display 201 for right eyes, and the display 202 for 
left eyes. 

The video camera 203 for right eyes, and the video camera 204 for left eyes. 

Each display 201,202 comprises an electrochromatic display and prism, and displays MR space 

image according to the observer's view position and sight line direction. 

[0043]The real space image photoed with the video camera 203 for right eyes is superimposed 
with the virtual space image which is in sight from the position of a right eye, and, thereby, MR 
space image for right eyes is generated. This MR space image for right eyes is displayed on the 
display 201 for right eyes. The real space image photoed with the video camera 204 for left eyes 
is superimposed with the virtual space image which is in sight from the position of a left eye, and, 
thereby, MR space image for left eyes is generated. This MR space image for left eyes is 
displayed on the display 202 for left eyes. Thus, if MR space image corresponding to each of the 
object for left eyes and the display 201,202 for right eyes is displayed, it is possible to make an 
observer do the stereo corporal vision of the MR space. The number of video cameras may be 
set to one, and a stereo corporal vision image may not be provided. 
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[0044]The head position direction sensor receiver 102 and the hand part position direction 
sensor receiver 103 can receive electromagnetic waves, an ultrasonic wave, etc. which are 
emitted from the position direction sensor transmitter 104, and can pinpoint the position 
direction of a sensor from the receiving intensity, phase, etc. It is fixed to the position in the 
space used as the place of a game, and this position direction sensor transmitter serves as a 
standard which detects the position and direction of each observer's head, and a hand part, 
respectively. 

[0045]Here r the head position direction sensor receiver 102 detects an observer's view position 
and a sight line direction. According to this embodiment, the head position direction sensor 
receiver 102 shall be fixed to HMD101. The hand part position direction sensor receiver 103 is 
what measures the position direction of a hand, For example, in compound real space 
(henceforth MR space), when required, it is equipped with the hand part position direction sensor 
receiver 103, and the position direction information on an observers hand does not need to 
equip — have an object on virtual space or a state changes according to a motion of a hand — 
when other. If the position information in each part of an observer s body is required, a required 
part will be equipped with a sensor receiver, respectively. 

[0046]To the observera [ 100 ],b [ 100 1 and 100c side. The above-mentioned position direction 
sensor transmitter 104, the loudspeaker 105, and the position direction sensor body 106 to 
which the head position direction sensor receiver 102, the hand position direction sensor 
receiver 103, and the position direction sensor transmitter 104 are connected, Each observers 
100a and 100b and the computer 107 which generates MR space image of every 100c are 
installed. Here, although the position direction sensor body 106 and the computer 107 approach 
a corresponding observer and are installed, they may be installed in the place distant from the 
observer. It is installed according to MR space which wants to generate two or more real space 
objects 1 10 united with MR space which each observers 100a, 100b, and 100c observe. Numbers 
arbitrary as this real space object 1 10 of objects can be installed. 

[0047]From the loudspeaker 105, the sound corresponding to the event which happened in MR 
space is generated. There are an explosion etc. which are generated as a sound corresponding to 
this event when the characters in virtual space collide, for example. The coordinates on MR 
space of the loudspeaker 105 are beforehand memorized by the system. And when the event 
which generates a certain sound happens, a sound occurs from the loudspeaker 105 arranged 
near [ where the event happened ] the MR space coordinates. For example, when an explosion 
event happens near the observer 100b, an explosion is emitted from the loudspeaker 105 near 
this observer 100b. The loudspeaker 105 is arranged in arbitrary numbers at arbitrary places so 
that proper presence may be given to an observer. 

[0048]It may be made to realize the sound effects in this system not using the method which 
arranges such a loudspeaker 105 but using HMD equipped with headphone. In this case, since 
only a specific observer can hear a sound, it is necessary to use the virtual source called 3D 
audio. 

[0049]The real space object 1 10 is a set which is specifically used for the attraction of an 
amusement facility. What kind of set is prepared changes by with what kind of MR space an 
observer is provided. 

[0050]Some colored thin wafers called the marker 120 are stuck on the surface of the real space 
object 1 10. The marker 120 is used in order for image processing to amend a gap of real space 
coordinates and virtual space coordinates. This amendment is mentioned later. 
[0051]Next, with reference to drawing 3 thru /or drawin g 6, **** and others is explained about 
the main points in MR space image generation displayed on HMD101. The figure showing the 
example of MR space image generation in case drawin g 3 has all the virtual space objects in this 
side, The figure and drawing 6 in which the example of MR space image generation in case the 
figure and drawin g 5 in which the example of MR space image generation in case drawi ng 4 does 
not use a penetration virtual space object is shown use a penetration virtual space object is 
shown are a figure showing the example of the gap amendment by the marker in the image 
generation system of drawing 1, 

[0052]MR space image as it let HMD101 pass to the observers 100a, 100b, and 100c and was 
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shown in drawin g 3, as if it is superimposed on a real space image and a virtual space image and 
the virtual space object existed in real space is displayed on real time. In generation of MR space 
image, in order to strengthen a feeling of fusion more, processing by MR space coordinates, the 
processing at the time of the lap by a penetration virtual body, and the gap compensation 
process by a marker are important. Each is explained below. 
[0053] First, MR space coordinates are explained, 

[0054]The virtual space object in which the observers 100a, 100b, and 100c were united in MR 
space, and interaction - for example, When CG character is struck by hand, a certain reaction is 
carried out to a character When generating MR space which performs - contact cannot be 
judged if another axis of coordinates was used in real space and virtual space. So, in this system, 
the coordinates of a real space object to unite with MR space and a virtual space object are 
changed into MR space-coordinates system, and all the objects are treated on this. 
[0055] About the real space object 1 10, an observer's view position united with MR space, a sight 
line direction, the position direction (measured value of a position direction sensor) of a hand, 
the position configuration information of the real space object 110, and other observers' position 
configuration information are changed into MR coordinate system. Similarly, about a virtual space 
object, the position configuration information of the virtual space object united with MR space is 
changed into MR coordinate system. Thus, by introducing MR space-coordinates system and 
changing a real space coordinate system and a virtual space coordinate system into MR space- 
coordinates system, the physical relationship and distance of a real space object and a virtual 
space object can be treated systematically, and it becomes possible to realize an interaction. 
[0056] Next, the problem at the time of a lap is explained. 

[0057]MR space image is generated by superimposing a real space image and a virtual space 
image, as shown in drawing 3 . Since in the case of dra wing 3 it sees from a viewpoint and all the 
virtual space objects exist to the front from a real space object in MR space coordinates, it is 
satisfactory. However, when a virtual space object exists in a depth direction from a real space 
object, as shown in dra wing 4 , the phenomenon in which a virtual space object is displayed 
before a real space object occurs. For this reason, it is necessary to perform processing for 
being hidden in the thing in this side about what is in a depth direction further from a viewpoint 
[ before superimposing ] about the coordinates in the inside of MR space of a realistic object 
and a virtual body. 

[0058]In order to perform the above-mentioned processing, when there is a real space object 
united with MR space, the virtual space object in which a background with the same shape, a 
position, and a direction is penetrated is beforehand defined as virtual space. For example, as 
shown in drawing 5 , a penetration virtual body is defined as virtual space with the same shape as 
three objects of real space. Thus, only a virtual body in back can be eliminated by using a 
penetration virtual space object, without overwriting a real image at the time of real image 
composition. 

[0059]Next, gap of real space coordinates and virtual space coordinates is explained. 
[0060]Although the position of a virtual space object and a direction are computed by 
calculation, since the real space object is measured by the position direction sensor, in this 
measured value, a certain error produces it. When this error generates MR space image, the 
position shift in MR space of a real space object and a virtual space object is made generated. 
[0061]Then, it is used in order that the above-mentioned marker 120 may amend such gap. In 
the real space united with MR space, the marker 120 is a wafer about [ with the combination of 
the specific color which does not exist all over the real space, or the specific color ] 2-5 cm 
around. 

[0062]It explains referring to drawing 6 for the procedure which amends the position shift of real 
space and virtual space using this marker 120. The coordinates on MR space of the marker 120 
who uses for amendment beforehand shall be defined as the system. 

[0063]As shown in drawin g 6, the view position and sight fine direction which were measured with 
the head position direction sensor receiver 102 are first changed into MR coordinate system, and 
a marker's image predicted from the view position in MR coordinate system and a sight line 
direction is created (F10). On the other hand, the image which extracted the marker position is 
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created from a real space image (F8 f F12). 

[0064]And these two images are compared, and the view position in the inside of MR space is 
assumed to be a right thing, and calculates the amount of gaps of the sight line direction in the 
inside of MR space (F14). By applying this computed amount of gaps to the sight line direction in 
MR space, the error produced between the real space objects of the virtual space object in MR 
space can be amended (F15, F16). In the above-mentioned example, the example at the time of 
not performing gap amendment by the marker 120 is shown in F17 in a figure by reference. 
[0065]Next, the hardware constitutions of the computer 107 which processes this system, a 
software configuration, and operation of each software are explained. 

[0066]First, it explains, referring to dr awing 7 f or the hardware constitutions of the computer 
107. Dra wi ng 7 is a hardware-constitutions figure of the computer 107 in the image generation 
system of drawmgj_. What is necessary is just to add this composition according to the number 
of observers, when increasing an observer. 

[0067]The computer 107 is a device which is formed in each observers 100a and 100b and every 
100c, and generates corresponding MR space image to drawi ng 1 so that it may be shown. To 
the computer 107. The video capture board 150 for right eyes, the video capture board 151 for 
left eyes, the graphics board 1 52 for right eyes, the graphics board 1 53 for left eyes, the sound 
board 158, the network interface 159, and the serial interface 154 are carried. Each apparatus is 
connected with CPU156, HDD(hard disk) 155, and the memory 157 via the bus in the inside of a 
computer, 

[0068]The video camera 203 for right eyes of HMD 101 is connected to the video capture board 
150 for right eyes, and the video camera 204 for left eyes is connected to the video capture 
board 151 for left eyes, respectively. The display 202 for left eyes is connected to the display 
201 for right eyes of HMD101, and the graphics board 153 for left eyes at the graphics board 152 
for right eyes, respectively. The loudspeaker 105 is connected to the sound board 158, and the 
network interface 159 is connected to the networks 130, such as LAN (Local Area Network). 
The position direction sensor body 106 is connected to the serial interface 154, and the head 
position direction sensor receiver 102, the hand part position direction sensor 103, and the 
position direction sensor transmitter 104 of HMD101 are connected to the position direction 
sensor body 106. 

[0069]The object for right eyes and the video capture board 150,151 for left eyes digitize the 
video signal from the object for right eyes, and the video camera 203,204 for left eyes, and 
incorporate it into the memory of the computer 107 by 30 frames per second 157. The 
incorporated real space image is superimposed with the virtual space image generated by 
computer 107, is outputted to the object for right eyes, and the graphics board 152,153 for left 
eyes, and is displayed on the object for right eyes, and the display 201,202 for left eyes. 
[0070]The position direction sensor body 106 with intensity, a phase, etc. of electromagnetic 
waves which the head position direction sensor receiver 1 02 and the hand part position direction 
sensor receiver 103 received. The position direction of each position direction sensor receiver 
102,103 is computed, and this computed position direction is notified to the computer 107 via 
the serial interface 154. 

[0071]The computer 108 (shown in drawin g 8) for performing each observers' 100a, 100b, and 
100c computer 107 and MR space status management mentioned later is connected to the 
network 130 connected to the network interface 159. 

[0072]The computer 107 corresponding to each observers 100a, 100b, and 100c shares the 
position of a view position, a sight line direction, and a virtual space object and direction which 
were detected by each observers 100a and 100b and every 100c via the network 130 by 
computer 108. Thereby, each computer 107 can generate MR space image to each observers 
100a, 100b, and 100c independently. 

[0073]When a playing event happens in MR space, a sound is emitted from the loudspeaker 105 
installed near the MR space coordinates which the playing event generated, but the command 
which computer performs also communicates via the network 130. 

[0074] Constitute from this embodiment so that special visual equipment, such as a three- 
dimensional converter, may not be used and one computer may be used to one test subject, but. 
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Make it one capture board which generates the Page Flip image according an input system to a 
three-dimensional converter, or, A power range system may be used as the graphics board of 
one 2 and you may have composition which starts a picture with the down converter 

after an Above & Below output with the graphics board of one sheet. 

[0075]Next, it explains, referring to d rawing 8 for the software configuration of the computer 107 
which processes this system, and each operation. Dra wing 8 is the software configuration figure 
carried in the computer of the image generation system of drawingjL 

[0076]In the computer 107, as shown in drawing 8 , each software of the position direction 
measurement software 320, the position amendment marker detection software 330, the sight 
line direction amendment software 350, the sound effect output software 340, and MR space 
image generation software 310 operates. Each software is stored in HDD155, is read from 
HDD155 by CPU156 and performed. 

[0077]Each observers' 100a, 100b, and 100c computer 107 is connected to the computer 108 via 
the network 130, and the MR space status management software 400 operates in the computer 
108. 

[0078]Although MR space status management software 400 is operated in this embodiment on 
computer 108 with the another computer 107 for MR space image generation formed to each 
observer, If there is no problem in throughput, it can also constitute so that MR space status 
management software 400 may be operated by computer 107. 

[0079]The position direction measurement software 320 communicates with the position 
direction sensor body 105, and measures the position of each position direction sensor receiver 
102,103, and a direction. And the line of sight position of the observer in MR space coordinates 
and a sight line direction are calculated from that measurement value, and this calculated value 
is notified to the sight line direction amendment software 350 with the position direction of the 
hand part position direction sensor 103. 

[0080]The gesture primary detecting element 321 in the position direction measurement 
software 320 detects the position direction of each position direction sensor 102,103 and the 
position direction relation of each position direction sensor 102,103, and an observers operation 
(gesture) guessed from a time shift, this — it is detected and gesture is notified to the sight line 
direction amendment software 350 if needed. 

[0081 ]The position amendment marker detection software 330 detects the marker 120 on a 
picture from the real space still picture transmitted from the real image acquisition part 312 of 
MR image generation software 310, and notifies the position on the picture to the sight line 
direction amendment software 350. 

[0082]The sight line direction amendment software 350 calculates the marker's 120 position in 
MR space image displayed in an observer's view position obtained from the position direction 
measurement software 320, a sight line direction to an observer's view position, and a sight line 
direction. This marker's predicted position is compared with the position in the inside of the 
picture of the actual marker detected with the position amendment marker detection software 
330, and a sight line direction is amended so that the position gap in the picture acquired by this 
comparison may occur. Thus, the sight line direction in the inside of amended MR space 
coordinates and a view position, the position direction of the hand part position sensing device 
receiver 103, and the gesture detected if required are notified to MR image generation software 
310. 

[0083]The sound effect output software 340 sounds a predetermined sound effect and 
background music (BGM) according to the performance command from MR image generation 
software 310 or MR space status management software 400. MR image generation software 310 
and MR status management software 400, The computer 107 to which the installed position and 
the loudspeaker 105 in MR space of the loudspeaker 105 are connected is recognized 
beforehand, When a certain playing event occurs on MR space, a sound is able to sound with the 
loudspeaker 105 near the position on MR space where the playing event happened. 
[0084] MR space status management software 400 manages the position direction and state of 
the position direction of all the real space objects and a state, and all the virtual space objects. 
About the position direction and state of a real space object, the position of the view position of 
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the observer concerned, a sight line direction, and the hand part position direction sensor 
receiver 103, a direction, and gesture are periodically notified from MR image generation 
software 310. Reception of these information is performed at any time, and it is not necessary to 
take timing in particular into consideration. About the position direction and state of a virtual 
space object, the virtual space status management department 401 in MR space status 
management software 400 notifies periodically. 

[0085]This MR space status management software 400 notifies these information to MR image 
generation software 310 which is operating by all the observers' computer 107 periodically. 
[0086]The virtual space status management department 401 manages and controls the matter 
about all the virtual space. The time of virtual space is made to pass and, specifically, it 
processes operating a virtual space object according to the scenario set up beforehand etc. The 
virtual space status management department 401 bears the role which advances the scenario 
corresponding to it, when an interaction and a gesture input (for example, a virtual space object 
is exploded when coordinates coincidence is carried out) with an observer, i.e., a real space 
object, and a virtual space object occur 

[0087]MR image generation software 310 generates MR space image for the observers 
concerned, and outputs it to each display 201,202 of HMD 101 of the observer concerned. This 
processing that carries out output Seki is that inside, and is divided by the state transmission 
and reception section 313, the virtual video generation part 311, the real image acquisition part 
312, and the image synthesis section 314. 

[0088]The state transmission and reception section 313 notifies periodically the position of the 
view position of the observer concerned notified from the sight line direction amendment 
software 350, a sight line direction, and the hand part position direction sensor receiver 103, a 
direction, and gesture to MR space status management software 400. The objective position 
direction and state which exist in all the MR space from MR space status management software 
400 are notified periodically. That is, about a real space object, the position of the view position 
of observers other than themselves, a sight line direction, and the hand part position sensing 
device receiver 103, a direction, and gesture are notified. About a virtual space object, the 
position direction and state of a virtual space object which the virtual space status management 
department 401 manages are notified. Reception of state information is possible always and does 
not need to take timing in particular into consideration. 

[0089]The virtual video generation part 31 1 generates the transparent virtual space image of a 
background seen from the view position of the observer concerned notified from the position 
direction, the state, and the sight line direction amendment software 350 of the virtual space 
object notified from MR space status management software 400, and the sight line direction. 
[0090]The real image acquisition part 312 incorporates a real space image from the object for 
right eyes, and the video capture board 150,151 for left eyes, stores it in the predetermined 
region of memory 157 or HDD155 (shown in drawing 7 ), and updating it is continued. 
[0091 ]The image synthesis section 314 reads the real space image which the above-mentioned 
portion generated from the memory 157 or HDD 155, is overlapped on a virtual space image, and 
is outputted to the display 201,202 of the observer concerned. 

[0092]Each observer can be provided with MR space image with the hardware and software 
which were explained above. 

[0093]As for MR space image which each observer looks at, since the state is managed in a 
unified manner with MR space status management software 400, it is possible to make it 
synchronize in time. 

[0094]Next, it explains, referring to drawing 9 thru/or drawin g 1 1 for the details of the operation 
of MR image generation software for reducing the time lag of a real space image and a virtual 
space image. 

[0095]Hardware and software relevant to generation of MR space image of the image generation 
system of dr awing 1 in draw ing 9, The figure showing the flow of the information typically, the 
time chart drawing 1 0 indicates the operation timing of MR image generation software of dr awing 
9^to be, and drawing 1 1 are flow charts which show operation of MR image generation software 
of drawin g 9. 
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[0096]If MR space state is notified to the state transmission and reception section 313 from MR 
space status management software 400 in MR image generation software 310 as shown in 
drawing 9 and drawing 1 1 (Step S100), Instructions of real space image drawing are published to 
the image synthesis section 314, and instructions of virtual space image generation are published 
to the virtual video generation part 312, respectively (A1, A10 which are shown in drawing 1 0). 
[0097]The image synthesis section 314 copies the newest real image data from the real image 
acquisition part 312 in response to instructions (A2 of drawing 10 ), and starts drawing on the 
memory 157 of the computer 107 (Step S102). 

[0098]It starts that the virtual video generation part 31 1 creates the position of the virtual body 
in virtual space, a direction, a state, a view position, and a sight line direction in the form of the 
state description form called scene graph based on the information which exists in the state 
receive section 313 (A11 of drawing 10; Step S104). 

[0099]Aithough he is trying to process Steps S102 and S104 in order, it may be made to process 
in parallel in this embodiment using multithread art. 

[0100]That real space image drawing processing of the image synthesis section 314 and the 
virtual space image generation processing of the virtual video generation part 312 end MR image 
generation software 310 (A12) And waiting (Step S106), After real space image drawing 
processing of this image synthesis section 314 and the virtual space image generation 
processing of the virtual video generation part 312 are completed, instructions of virtual space 
image drawing are gone into effect in the image synthesis section 314. 

[0101]The image synthesis section 314 investigates whether an observers view position and the 
sight line direction are updated (Step S107), Here, when renewal of the above-mentioned 
information is performed, the newest view position and a sight line direction are acquired (Step 
S108), and the virtual space image which is in sight from this newest view position and a sight 
line direction is drawn (Step S1 10). The processing which changes this view position and a sight 
line direction, and draws is so small that it can be disregarded compared with the whole drawing 
time f and does not become a problem. When the above-mentioned information is not updated, 
the above-mentioned step S108 and S1 10 are skipped, and drawing of a virtual space image is 
continued as it is. 

[01 02] And the image synthesis section 314 compounds the real space image and virtual space 
image which drew, and outputs them to the display 10. (Step S1 12) 

A series of MR space image generation processings are completed above. Here, this processing 
will be ended, if it investigated whether the end instruction was received and popularity is won. 
The above processing is repeated if popularity is not won (Step S114). 
[0103]In the midst of a series of above-mentioned processings, the state transmission and 
reception section 314 may receive a new state from MR space status management software 400. 
in this case, it is shown in dra wing 10 — as — AV, A10\ and A1 — although that is notified like 
" t A10'\ these are disregarded until the image synthesis section 314 investigates whether there 
was any notice at the above-mentioned step S100. 

[0104]Thus, since the state of MR space image which each observer looks at is managed in a 
unified manner with MR space status management software 400, a real space image and a virtual 
space image can be synchronized in time. When two or more observers use this system 
simultaneously by this, it enables each observer to see the image of the same time. It becomes 
possible to use an observer's newest position direction information, and time gap of a real space 
image and a virtual space image can be reduced. As a display which provides an observer with 
MR space image especially, when HMD101 is used, a response when an observer shakes a head 
at right and left improves. 

[0105]As mentioned above, in this system, the time lag of a real space image and a virtual space 
image can be reduced, and an observer can be provided with MR space image which has a feeling 
of devotion more. 

[0106]The storage which recorded the program code of the software which realizes the function 
(the flow chart shown in dr awing 1 1 is included) of an above-mentioned embodiment It cannot 
be overemphasized that it is attained, also when a system or a device is supplied and the 
computer (or CPU and MPU) of the system or a device reads and executes the program code 
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stored in the storage. 

[0107]In this case, the function of an embodiment which the program code itself read from the 
storage mentioned above will be realized, and the storage which memorized that program code 
will constitute this invention. 

[Q108]As a storage for supplying a program code, a floppy disk, a hard disk, an optical disc, a 
magneto-optical disc, CD-ROM, CD-R, magnetic tape, a nonvolatile memory card, ROM, DVD- 
ROM, etc. can be used, for example. 

[0109]By executing the program code which the computer read, Based on directions of the 
program code the function of an embodiment mentioned above is not only realized, but, It cannot 
be overemphasized that it is contained also when the function of an embodiment which 
performed a part or all of processing that OS (operating system) etc. which are working on a 
computer are actual, and was mentioned above by the processing is realized. 
[01 10] After the program code read from the storage was written in the memory with which the 
function expansion unit connected to the expansion board inserted in the computer or the 
computer is equipped, It cannot be overemphasized that it is contained also when the function of 
an embodiment which performed a part or all of processing that CPU etc. with which the 
expansion board and function expansion unit are equipped are actual, based on directions of the 
program code, and was mentioned above by the processing is realized. 
[0111] 

[Effect of the Invention]The imaging means which picturizes the real space in the sight line 
direction in an observers view position according to the image generation system of this 
invention as explained above, A detection means to detect an observer's view position and sight 
line direction, and the virtual space video generating means which generates the virtual space 
image in an observer's view position and sight line direction which were detected by the 
detection means, The compound video generating means which generates the compound image 
which compounded the virtual space image generated by the virtual space video generating 
means and the real space image outputted from the imaging means, Since it has a management 
tool which manages in common the information on the object which exists in each and its 
position, and a direction to the displaying means which displays the compound image generated 
by the compound video generating means, and real space and virtual space, The time lag of a 
real space image and a virtual space image can be reduced, and an observer can be provided 
with the compound image which has a feeling of devotion more. 

[01 12]The process of detecting an observer's view position and line of sight position according 
to the image generation generation method of this invention, The process of acquiring the real 
space image in an observer's view position and sight line direction, The process of obtaining 
management information including the object which exists in each of real space and virtual space 
and its position, and a direction, Since it has the process of generating the virtual space image in 
an observer's view position and sight line direction, and the process of generating the compound 
image which compounded the virtual space image and the real space image according to 
management information, according to management information, The time lag of a real space 
image and a virtual space image can be reduced, and an observer can be provided with the 
compound image which has a feeling of devotion more. 

[01 13]The detecting module which detects an observer's view position and sight line direction 
where the program was detected by the detection means according to the storage of this 
invention, The virtual space image generating module for generating the virtual space image from 
an observer's view position and sight line direction detected with the detecting module, The 
compound image generating module for generating a compound image from the virtual space 
image and real space image which were generated with the virtual space image generating 
module, Since it has an administrative module for managing the object which exists in each of 
real space and virtual space and its position, and a direction, the time lag of a real space image 
and a virtual space image can be reduced, and an observer can be provided with the compound 
image which has a feeling of devotion more. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



TECHNICAL FIELD 



[Field of the Invention]This invention compounds the real space image and virtual space images, 
such as computer graphics, which were incorporated from the photographing device of a video 
camera etc., and relates to the image generation system, the image generation method, and 
storage which generate a compound image. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation, 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



PRIOR ART 



[Description of the Prior Art]As a mixed reality device used as a display, the conventional HMD 
(head mount display), There are a proposal, now a thing which is by Oshima, Sato, Yamamoto, 
Tamura, etc, (Oshima, Sato, Yamamoto, and Tamura: references, such as "realization of an AR2 
hockey:cooperation type mixed reality system", a Virtual Reality Society of Japan paper 
magazine, Vol.3, No.2, pp.55-60, and 1998 J. 



[Translation done,] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer So the translation may not reflect the original 
precisely. 

2, **** shows the word which can not be translated. 
3in the drawings, any words are not translated. 



EFFECT OF THE INVENTION 



[Effect of the Invention]The imaging means which picturizes the real space in the sight line 
direction in an observer's view position according to the image generation system of this 
invention as explained above, A detection means to detect an observer s view position and sight 
line direction, and the virtual space video generating means which generates the virtual space 
image in an observer s view position and sight line direction which were detected by the 
detection means, The compound video generating means which generates the compound image 
which compounded the virtual space image generated by the virtual space video generating 
means and the real space image outputted from the imaging means, Since it has a management 
tool which manages in common the information on the object which exists in each and its 
position, and a direction to the displaying means which displays the compound image generated 
by the compound video generating means, and real space and virtual space, The time lag of a 
real space image and a virtual space image can be reduced, and an observer can be provided 
with the compound image which has a feeling of devotion more. 

[01 12]The process of detecting an observer's view position and line of sight position according 
to the image generation generation method of this invention, The process of acquiring the real 
space image in an observer s view position and sight line direction, The process of obtaining 
management information including the object which exists in each of real space and virtual space 
and its position, and a direction, Since it has the process of generating the virtual space image in 
an observer's view position and sight line direction, and the process of generating the compound 
image which compounded the virtual space image and the real space image according to 
management information, according to management information, The time lag of a real space 
image and a virtual space image can be reduced, and an observer can be provided with the 
compound image which has a feeling of devotion more. 

[0113]The detecting module which detects an observer's view position and sight line direction 
where the program was detected by the detection means according to the storage of this 
invention, The virtual space image generating module for generating the virtual space image from 
an observers view position and sight line direction detected with the detecting module, The 
compound image generating module for generating a compound image from the virtual space 
image and real space image which were generated with the virtual space image generating 
module, Since it has an administrative module for managing the object which exists in each of 
real space and virtual space and its position, and a direction, the time lag of a real space image 
and a virtual space image can be reduced, and an observer can be provided with the compound 
image which has a feeling of devotion more. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
3Jn the drawings, any words are not translated. 



TECHNICAL PROBLEM 



[Problem(s) to be Solved by the Invention] However, in the above-mentioned conventional device, 
if an observer shakes a head at right and left, a real space image will follow in footsteps 
immediately, but phenomena, like a virtual space image follows later than a real space image are 
checked. That is, a time gap of a real space image and a virtual space image is conspicuous. 
[0004]The purpose of this invention reduces the time lag of a real space image and a virtual 
space image, and there is in providing the image generation system, the image generation 
method, and storage which can provide an observer with the compound image which has a 
feeling of devotion more. 



[Translation done.] 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1This document has been translated by computer So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



MEANS 



[Means for Solving the Problem]An imaging means in which the invention according to claim 1 
picturizes real space in the sight line direction in an observers view position, A detection means 
to detect said observer's view position and a sight line direction, and a virtual space video 
generating means which generates a virtual space image in an observer's view position and a 
sight line direction which were detected by said detection means, A compound video generating 
means which generates a compound image which compounded a virtual space image generated 
by said virtual space video generating means and a real space image outputted from said imaging 
means, It has a management tool which manages in common information on an object which 
exists in each and its position, and a direction to a displaying means which displays a compound 
image generated by said compound video generating means, and real space and virtual space. 
[0006]The invention according to claim 2 is characterized by the ability of said management tool 
to update each object of said real space and said virtual space, its position, a direction, and 
information on a state in the image generation system according to claim 1. 

[0007]In the invention according to claim 3, in the image generation system according to claim 2, 
said management tool notifies each object of said real space and said virtual space, its position, 
a direction, and information on a state to said compound video generating means for every 
predetermined time. 

[0008]In the image generation system according to claim 3, said virtual video generating means 
presupposes that the invention according to claim 4 is responded to an information update from 
said management tool, and it generates a virtual space image based on each object of said real 
space and said virtual space, its position, and a direction. 

[0009]In the invention according to claim 5, in the image generation system according to claim 3 
or 4, said compound video generating means starts drawing of said real space image according to 
an information update from said management tool. 

[0010]In the image generation system according to claim 5, as for the invention according to 
claim 6, said compound video generating means performs composition with said real space image 
and said virtual space image, after it ends drawing of said real space image and a virtual space 
image is generated by said virtual space video generating means. 

[001 1]In the image generation system according to claim 6, the invention according to claim 7 
said compound video generating means, Just before performing composition of said real space 
image and said virtual space image, regeneration of said virtual space image is performed based 
on an observer's view position and a sight line direction which were detected by said detection 
means, 

[0012]In the image generation system according to claim 3, the invention according to claim 8 
said virtual space video generating means, According to an information update from said 
management tool, each object of said real space and said virtual space, Perform processing 
which generates a virtual space image based on the position and direction, and said compound 
video generating means, According to an information notification from said management tool, 
processing which starts drawing of said real space image is arranged in parallel with generation 
processing of said virtual space image by said virtual space video generating means, and is 
performed. 
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[0013]The invention according to claim 9 is characterized by the number of said observers being 
plurality in an image generation system of any one statement of claim 1 thru/or 8. 
[0014]The invention according to claim 10 has a motion detecting means which detects 
operation including an observer's gesture made applicable to detection, a state, etc. based on a 
detection result of said detection means in an image generation system of any one statement of 
claim 1 thru/or 9. 

[0015]The invention according to claim 1 1 is characterized by it being possible to use an 
observer's operation detected by said motion detecting means as an influence input to an object 
which exists in said compound real space and this compound real space in the image generation 
system according to claim 10. 

[001 6] A process of the invention according to claim 12 being the image generation method for 
compounding a virtual space image on a real space image in the sight line direction in an 
observer s view position, and generating a compound image, and detecting said observer's view 
position and a line of sight position, A process of acquiring a real space image in said observer's 
view position and a sight line direction, A process of obtaining management information including 
an object which exists in each of real space and virtual space and its position, and a direction, 
According to said management information, it has a process of generating a virtual space image 
in said observer s view position and a sight line direction, and the process of generating a 
compound image which compounded said virtual space image and said real space image 
according to said management information. 

[0017]The invention according to claim 13 has the process of updating said management 
information, in an image generation method according to claim 12. 

[0018]As for the invention according to claim 14, in an image generation method according to 
claim 12 or 13, said management information is notified to a process of generating sard 
compound image for every predetermined time. 

[0019]As for the invention according to claim 15, in an image generation method according to 
claim 14, generation of said virtual space image is generated based on each object of said real 
space and said virtual space, its position, and a direction according to renewal of said 
management information. 

[0020] As for the invention according to claim 16, in an image generation method according to 
claim 14 or 15, drawing of said acquired real space image is started by generate time of said 
compound image ignited by renewal of said management information. 

[0021 ]In an image generation method according to claim 16, the invention according to claim 17 
generation of said compound image, After drawing of said real space image is completed and a 
virtual space image is generated by said virtual space image generation process, it is carried out 
by compounding said real space image and said virtual space image. 

[0022] Just before the invention according to claim 18 performs composition of said real space 
image and said virtual space image in an image generation method according to claim 17, 
regeneration of said virtual space image is performed based on a view position and a sight line 
direction of said detected observer. 

[0023]In an image generation method according to claim 14 the invention according to claim 19, 
According to renewal of said management information, generation of a virtual space image based 
on said management information is started, drawing of said acquired real space image 
accompanying generation of said compound image is started, and it is carried out by generation 
of said virtual space image and drawing of said real space image standing in a row. 
[0024]The invention according to claim 20 is characterized by the number of said observers 
being plurality in an image generation method of any one statement of claim 12 thru/or 19. 
[0025]The invention according to claim 21 has the process of detecting operation including an 
observers gesture made applicable to detection based on said observer s view position and a 
line of sight position, a state, etc., in an image generation method according to claim 20. 
[0026]The invention according to claim 22 is characterized by it being possible to use said an 
observers detected operation as an influence input to an object which exists in space of said 
compound image and space of this compound image in an image generation method according to 
claim 21. 
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[0027]An imaging means which picturizes a real space image [ in / this invention / in an 
observer s view position / in the invention according to claim 23 / the sight line direction ], An 
image generation system provided with a detection means to detect said observer's view 
position and a sight iine direction, and a displaying means for displaying a compound image which 
compounded a virtual space image in said observer's view position and a sight line direction to 
said real space image is characterized by comprising: 

A detecting module which is the storage which stored a program for generating said compound 

image, and in which computer reading is possible, and detects an observer s view position and a 

sight line direction where said program was detected by said detection means. 

A virtual space image generating module for generating a virtual space image from an observers 

view position and a sight line direction detected with said detecting module, 

A compound image generating module for generating said compound image from a virtual space 

image generated with said virtual space image generating module, and said real space image. 

An administrative module for managing an object which exists in each of real space and virtual 

space and its position, and a direction. 

[0028]The invention according to claim 24 is characterized by the ability of said administrative 
module to perform renewal of each object of said real space and said virtual space, its position, a 
direction, and a state in the storage according to claim 23, 

[0029]In the invention according to claim 25, in the storage according to claim 24 T said 
administrative module notifies each object of said real space and said virtual space, its position, 
a direction, and a state to said compound video generating means for every predetermined time. 
[0030]In the storage according to claim 25, the invention according to claim 26 said virtual space 
image generating module, A virtual space image based on each object of said real space and said 
virtual space, its position, and a direction is generated ignited by an information update from said 
administrative module. 

[0031 ]In the invention according to claim 27, in the storage according to claim 25 or 26, said 
compound image generating module starts drawing of said real space image ignited by an 
information update from said administrative module. 

[0032]In the storage according to claim 27, the invention according to claim 28 said compound 
image generating module, After it ends drawing of said real space image and a virtual space 
image is generated by said virtual space image generating module, composition with said real 
space image and said virtual space image is performed. 

[0033]In the storage according to claim 28, the invention according to claim 29 said compound 
image generating module, Just before performing composition of said real space image and said 
virtual space image, regeneration of said virtual space image is performed based on an observer s 
view position and a sight line direction which were detected with said detecting module. 
[0034]In the storage according to claim 25, the invention according to claim 30 said virtual space 
image generating module, Ignited by an information update from said administrative module, each 
object of said real space and said virtual space, Perform processing which generates a virtual 
space image based on the position and direction, and said compound image generating module, 
According to an information notification from said administrative module, processing which starts 
drawing of said real space image is arranged in parallel with generation processing of said virtual 
space image by said virtual space image generating module, and is performed. 
[0035]The invention according to claim 31 is characterized by the number of said observers 
being plurality in a storage of any one statement of claim 23 thru/or 30. 

[0036]The invention according to claim 32 has a detecting module of operation which detects 
operation including a gesture of an observer who makes said program applicable to detection 
based on a detection result of said detecting module, a state, etc. in a storage of any one 
statement of claim 23 thru/or 31. 

[0037]The invention according to claim 33 is characterized by it being possible to use an 
observer s operation detected with said detecting module of operation as influence on an object 
which exists in said compound image space and this compound image space in the storage 
according to claim 32. 
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[0038] 

[Embodiment of the Invention] Hereafter, it explains, referring to drawings for an embodiment of 
the invention. 

[0039]The appearance lineblock diagram of the image generation system which displays the 
compound image which drawing 1 requires for one gestatt of operation of this invention, and 
drawing 2 are the perspective views showing the composition of HMD with which the head of the 
observer of drawin g 1 is equipped. 

[0040]ln this embodiment, as shown in drawin g 1, a system shall be installed in the interior of a 
room about 5 m around, and the three observers 100a, 100b, and 100c shall experience mixed 
reality by this system. The setting position of this system, a size, and the number of observers 
are not limited to this, and can be changed freely. 

[0041]Each observers 100a, 100b, and 100c A compound real space image. HMD(head mount 
display)101 for providing each observer with (it is hereafter called MR space image) on a head. A 
right hand (for example, dominant hand) is equipped with the hand part position direction sensor 
receiver 103 for detecting the position and direction of a hand for the head position direction 
sensor receiver 102 which detects the position and direction of a head on a head, respectively. 
[0042]This HMD 101 is provided with the following. 

As shown in drawin g 2 (a) and (b), they are the display 201 for right eyes, and the display 202 for 
left eyes. 

The video camera 203 for right eyes, and the video camera 204 for left eyes. 

Each display 201,202 comprises an electrochromatic display and prism, and displays MR space 

image according to the observer s view position and sight line direction, 

[0043]The real space image photoed with the video camera 203 for right eyes is superimposed 
with the virtual space image which is in sight from the position of a right eye, and, thereby, MR 
space image for right eyes is generated. This MR space image for right eyes is displayed on the 
display 201 for right eyes. The real space image photoed with the video camera 204 for left eyes 
is superimposed with the virtual space image which is in sight from the position of a left eye, and, 
thereby, MR space image for left eyes is generated. This MR space image for left eyes is 
displayed on the display 202 for left eyes. Thus, if MR space image corresponding to each of the 
object for left eyes and the display 201,202 for right eyes is displayed, it is possible to make an 
observer do the stereo corporal vision of the MR space. The number of video cameras may be 
set to one, and a stereo corporal vision image may not be provided. 

[0044]The head position direction sensor receiver 102 and the hand part position direction 
sensor receiver 103 can receive electromagnetic waves, an ultrasonic wave, etc. which are 
emitted from the position direction sensor transmitter 104, and can pinpoint the position 
direction of a sensor from the receiving intensity, phase, etc. It is fixed to the position in the 
space used as the place of a game, and this position direction sensor transmitter serves as a 
standard which detects the position and direction of each observer s head, and a hand part, 
respectively. 

[0045]Here, the head position direction sensor receiver 102 detects an observer s view position 
and a sight line direction. According to this embodiment, the head position direction sensor 
receiver 102 shall be fixed to HMD 101. The hand part position direction sensor receiver 103 is 
what measures the position direction of a hand, For example, in compound real space 
(henceforth MR space), when required, it is equipped with the hand part position direction sensor 
receiver 103, and the position direction information on an observers hand does not need to 
equip — have an object on virtual space or a state changes according to a motion of a hand — 
when other. If the position information in each part of an observer's body is required, a required 
part will be equipped with a sensor receiver, respectively, 

[0046]To the observera [ 100 ],b [ 100 3, and 100c side. The above-mentioned position direction 
sensor transmitter 104, the loudspeaker 105, and the position direction sensor body 106 to 
which the head position direction sensor receiver 102, the hand position direction sensor 
receiver 103, and the position direction sensor transmitter 104 are connected, Each observers 
100a and 100b and the computer 107 which generates MR space image of every 100c are 
installed. Here, although the position direction sensor body 106 and the computer 107 approach 
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a corresponding observer and are installed, they may be installed En the place distant from the 
observer. It is installed according to MR space which wants to generate two or more real space 
objects 110 united with MR space which each observers 100a, 100b, and 100c observe. Numbers 
arbitrary as this real space object 1 10 of objects can be installed. 

[0047] From the loudspeaker 105, the sound corresponding to the event which happened in MR 
space is generated. There are an explosion etc. which are generated as a sound corresponding to 
this event when the characters in virtual space collide, for example. The coordinates on MR 
space of the loudspeaker 105 are beforehand memorized by the system. And when the event 
which generates a certain sound happens, a sound occurs from the loudspeaker 105 arranged 
near [ where the event happened ] the MR space coordinates. For example, when an explosion 
event happens near the observer 100b, an explosion is emitted from the loudspeaker 105 near 
this observer 100b. The loudspeaker 105 is arranged in arbitrary numbers at arbitrary places so 
that proper presence may be given to an observer, 

[0048]It may be made to realize the sound effects in this system not using the method which 
arranges such a loudspeaker 105 but using HMD equipped with headphone. In this case, since 
only a specific observer can hear a sound, it is necessary to use the virtual source called 3D 
audio. 

[0049]The real space object 1 10 is a set which is specifically used for the attraction of an 
amusement facility. What kind of set is prepared changes by with what kind of MR space an 
observer is provided. 

[0050]Some colored thin wafers called the marker 120 are stuck on the surface of the real space 
object 1 10. The marker 120 is used in order for image processing to amend a gap of real space 
coordinates and virtual space coordinates. This amendment is mentioned later. 
[0051]Next, with reference to drawin g 3 thru/or drawing 6, **** and others is explained about 
the main points in MR space image generation displayed on HMD101. The figure showing the 
example of MR space image generation in case drawi ng 3 has all the virtual space objects in this 
side, The figure and drawing 6 in which the example of MR space image generation in case the 
figure and drawing 5 in which the example of MR space image generation in case dr awing 4 does 
not use a penetration virtual space object is shown use a penetration virtual space object is 
shown are a figure showing the example of the gap amendment by the marker in the image 
generation system of dra wing 1 . 

[0052] MR space image as it let HMD 101 pass to the observers 100a, 100b, and 100c and was 
shown in draw ing 3, as if it is superimposed on a real space image and a virtual space image and 
the virtual space object existed in real space is displayed on real time. In generation of MR space 
image, in order to strengthen a feeling of fusion more, processing by MR space coordinates, the 
processing at the time of the lap by a penetration virtual body, and the gap compensation 
process by a marker are important. Each is explained below. 
[0053] First, MR space coordinates are explained. 

[0054]The virtual space object in which the observers 100a, 100b, and 100c were united in MR 
space, and interaction - for example, When CG character is struck by hand, a certain reaction is 
carried out to a character. When generating MR space which performs - contact cannot be 
judged if another axis of coordinates was used in real space and virtual space. So, in this system, 
the coordinates of a real space object to unite with MR space and a virtual space object are 
changed into MR space-coordinates system, and all the objects are treated on this. 
[0055]About the real space object 110, an observer's view position united with MR space, a sight 
line direction, the position direction (measured value of a position direction sensor) of a hand, 
the position configuration information of the real space object 110, and other observers' position 
configuration information are changed into MR coordinate system. Similarly, about a virtual space 
object the position configuration information of the virtual space object united with MR space is 
changed into MR coordinate system. Thus, by introducing MR space-coordinates system and 
changing a real space coordinate system and a virtual space coordinate system into MR space- 
coordinates system, the physical relationship and distance of a real space object and a virtual 
space object can be treated systematically, and it becomes possible to realize an interaction. 
[0056]Next, the problem at the time of a lap is explained. 
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[0057]MR space image is generated by superimposing a real space image and a virtual space 
image, as shown in drawing 3. Since in the case of d rawing 3 it sees from a viewpoint and all the 
virtual space objects exist to the front from a real space object in MR space coordinates, it is 
satisfactory. However, when a virtual space object exists in a depth direction from a real space 
object, as shown in dr awing 4 , the phenomenon in which a virtual space object is displayed 
before a real space object occurs. For this reason, it is necessary to perform processing for 
being hidden in the thing in this side about what is in a depth direction further from a viewpoint 
[ before superimposing ] about the coordinates in the inside of MR space of a realistic object 
and a virtual body, 

[0058]ln order to perform the above-mentioned processing, when there is a real space object 
united with MR space, the virtual space object in which a background with the same shape, a 
position, and a direction is penetrated is beforehand defined as virtual space. For example, as 
shown in drawing 5 , a penetration virtual body is defined as virtual space with the same shape as 
three objects of real space. Thus, only a virtual body in back can be eliminated by using a 
penetration virtual space object, without overwriting a real image at the time of real image 
composition. 

[0059] Next, gap of real space coordinates and virtual space coordinates is explained. 
[0060]Although the position of a virtual space object and a direction are computed by 
calculation, since the real space object is measured by the position direction sensor, in this 
measured value, a certain error produces it. When this error generates MR space image, the 
position shift in MR space of a real space object and a virtual space object is made generated, 
[0061]Then, it is used in order that the above-mentioned marker 120 may amend such gap. In 
the real space united with MR space, the marker 120 is a wafer about [ with the combination of 
the specific color which does not exist all over the real space, or the specific color ] 2-5 cm 
around, 

[0062]It explains referring to drawing 6 for the procedure which amends the position shift of real 
space and virtual space using this marker 120. The coordinates on MR space of the marker 120 
who uses for amendment beforehand shall be defined as the system. 

[0063]As shown in drawing 6 , the view position and sight line direction which were measured with 
the head position direction sensor receiver 102 are first changed into MR coordinate system, and 
a marker s image predicted from the view position in MR coordinate system and a sight line 
direction is created (F10). On the other hand, the image which extracted the marker position is 
created from a real space image (F8, F12). 

[0064]And these two images are compared, and the view position in the inside of MR space is 
assumed to be a right thing, and calculates the amount of gaps of the sight line direction in the 
inside of MR space (F14). By applying this computed amount of gaps to the sight line direction in 
MR space, the error produced between the real space objects of the virtual space object in MR 
space can be amended (F15, F16). In the above-mentioned example, the example at the time of 
not performing gap amendment by the marker 120 is shown in F17 in a figure by reference. 
[0065]Next, the hardware constitutions of the computer 107 which processes this system, a 
software configuration, and operation of each software are explained. 

[0066]First, it explains, referring to drawing 7 for the hardware constitutions of the computer 
107. Drawing 7 is a hardware-constitutions figure of the computer 107 in the image generation 
system of drawing 1 . What is necessary is just to add this composition according to the number 
of observers, when increasing an observer. 

[0067]The computer 107 is a device which is formed in each observers 100a and 100b and every 
100c r and generates corresponding MR space image to dr awing 1 so that it may be shown. To 
the computer 107. The video capture board 150 for right eyes, the video capture board 151 for 
left eyes, the graphics board 152 for right eyes, the graphics board 153 for left eyes, the sound 
board 158, the network interface 159, and the serial interface 154 are carried. Each apparatus is 
connected with CPU156, HDDfliard disk) 155, and the memory 157 via the bus in the inside of a 
computer. 

[0068]The video camera 203 for right eyes of HMD101 is connected to the video capture board 
150 for right eyes, and the video camera 204 for left eyes is connected to the video capture 
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board 151 for left eyes, respectively. The display 202 for left eyes is connected to the display 
201 for right eyes of HMD 101, and the graphics board 153 for left eyes at the graphics board 152 
for right eyes, respectively. The loudspeaker 105 is connected to the sound board 158, and the 
network interface 159 is connected to the networks 130, such as LAN (Local Area Network). 
The position direction sensor body 106 is connected to the serial interface 154, and the head 
position direction sensor receiver 102, the hand part position direction sensor 103, and the 
position direction sensor transmitter 104 of HMD101 are connected to the position direction 
sensor body 106. 

[0069]The object for right eyes and the video capture board 150,151 for left eyes digitize the 
video signal from the object for right eyes, and the video camera 203,204 for left eyes, and 
incorporate it into the memory of the computer 107 by 30 frames per second 157. The 
incorporated real space image is superimposed with the virtual space image generated by 
computer 107, is outputted to the object for right eyes, and the graphics board 152,153 for left 
eyes, and is displayed on the object for right eyes, and the display 201,202 for left eyes. 
[0070]The position direction sensor body 106 with intensity, a phase, etc. of electromagnetic 
waves which the head position direction sensor receiver 102 and the hand part position direction 
sensor receiver 1 03 received. The position direction of each position direction sensor receiver 
102,103 is computed, and this computed position direction is notified to the computer 107 via 
the serial interface 154, 

[0071]The computer 108 (shown in drawing 8) for performing each observers' 100a, 100b, and 
100c computer 107 and MR space status management mentioned later is connected to the 
network 130 connected to the network interface 159. 

[0072]The computer 107 corresponding to each observers 100a, 100b, and 100c shares the 
position of a view position, a sight line direction, and a virtual space object and direction which 
were detected by each observers 100a and 100b and every 100c via the network 130 by 
computer 108. Thereby, each computer 107 can generate MR space image to each observers 
100a, 100b, and 100c independently. 

[0073]When a playing event happens in MR space, a sound is emitted from the loudspeaker 105 
installed near the MR space coordinates which the playing event generated, but the command 
which computer performs also communicates via the network 130. 

[0074]Constitute from this embodiment so that special visual equipment, such as a three- 
dimensional converter, may not be used and one computer may be used to one test subject, but. 
Make it one capture board which generates the Page Flip image according an input system to a 
three-dimensional converter, or, A power range system may be used as the graphics board of 
one 2 and you may have composition which starts a picture with the down converter 

after an Above & Below output with the graphics board of one sheet. 

[0075]Next, it explains, referring to drawing 8 for the software configuration of the computer 107 
which processes this system, and each operation. Drawing 8 is the software configuration figure 
carried in the computer of the image generation system of drawing 1 . 

[0076]In the computer 107, as shown in dra wing 8 , each software of the position direction 
measurement software 320, the position amendment marker detection software 330, the sight 
line direction amendment software 350, the sound effect output software 340, and MR space 
image generation software 310 operates. Each software is stored in HDD 155, is read from 
HDD 155 by GPU 156 and performed. 

[0077]Each observers' 100a, 100b, and 100c computer 107 is connected to the computer 108 via 
the network 130, and the MR space status management software 400 operates in the computer 
108. 

[0078]Although MR space status management software 400 is operated in this embodiment on 
computer 108 with the another computer 107 for MR space image generation formed to each 
observer, If there is no problem in throughput, it can also constitute so that MR space status 
management software 400 may be operated by computer 107. 

[0079]The position direction measurement software 320 communicates with the position 
direction sensor body 105, and measures the position of each position direction sensor receiver 
102,103, and a direction. And the line of sight position of the observer in MR space coordinates 
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and a sight line direction are calculated from that measurement value, and this calculated value 
is notified to the sight line direction amendment software 350 with the position direction of the 
hand part position direction sensor 103. 

[0080]The gesture primary detecting element 321 in the position direction measurement 
software 320 detects the position direction of each position direction sensor 102,103 and the 
position direction relation of each position direction sensor 102,103, and an observers operation 
(gesture) guessed from a time shift, this — it is detected and gesture is notified to the sight line 
direction amendment software 350 if needed. 

[0081 jThe position amendment marker detection software 330 detects the marker 120 on a 
picture from the real space still picture transmitted from the real image acquisition part 312 of 
MR image generation software 310, and notifies the position on the picture to the sight line 
direction amendment software 350. 

[0082]The sight line direction amendment software 350 calculates the markers 120 position in 
MR space image displayed in an observer's view position obtained from the position direction 
measurement software 320, a sight line direction to an observer's view position, and a sight line 
direction. This markers predicted position is compared with the position in the inside of the 
picture of the actual marker detected with the position amendment marker detection software 
330, and a sight line direction is amended so that the position gap in the picture acquired by this 
comparison may occur. Thus, the sight line direction in the inside of amended MR space 
coordinates and a view position, the position direction of the hand part position sensing device 
receiver 103, and the gesture detected if required are notified to MR image generation software 
310. 

[0083]The sound effect output software 340 sounds a predetermined sound effect and 
background music (BGM) according to the performance command from MR image generation 
software 310 or MR space status management software 400. MR image generation software 310 
and MR status management software 400, The computer 107 to which the installed position and 
the loudspeaker 105 in MR space of the loudspeaker 105 are connected is recognized 
beforehand, When a certain playing event occurs on MR space, a sound is able to sound with the 
loudspeaker 105 near the position on MR space where the playing event happened, 
[0084] MR space status management software 400 manages the position direction and state of 
the position direction of all the real space objects and a state, and all the virtual space objects. 
About the position direction and state of a real space object, the position of the view position of 
the observer concerned, a sight line direction, and the hand part position direction sensor 
receiver 103, a direction, and gesture are periodically notified from MR image generation 
software 310. Reception of these information is performed at any time, and it is not necessary to 
take timing in particular into consideration. About the position direction and state of a virtual 
space object, the virtual space status management department 401 in MR space status 
management software 400 notifies periodically. 

[0085]This MR space status management software 400 notifies these information to MR image 
generation software 310 which is operating by all the observers' computer 107 periodically. 
[0086]The virtual space status management department 401 manages and controls the matter 
about all the virtual space. The time of virtual space is made to pass and, specifically, it 
processes operating a virtual space object according to the scenario set up beforehand etc. The 
virtual space status management department 401 bears the role which advances the scenario 
corresponding to it, when an interaction and a gesture input (for example, a virtual space object 
is exploded when coordinates coincidence is carried out) with an observer, i.e., a real space 
object, and a virtual space object occur. 

[0087]MR image generation software 310 generates MR space image for the observers 
concerned, and outputs it to each display 201,202 of HMD101 of the observer concerned. This 
processing that carries out output Seki is that inside, and is divided by the state transmission 
and reception section 313, the virtual video generation part 31 1, the real image acquisition part 
312, and the image synthesis section 314. 

[0088]The state transmission and reception section 313 notifies periodically the position of the 
view position of the observer concerned notified from the sight line direction amendment 
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software 350, a sight line direction, and the hand part position direction sensor receiver 103, a 
direction, and gesture to MR space status management software 400. The objective position 
direction and state which exist in all the MR space from MR space status management software 
400 are notified periodically. That is, about a real space object, the position of the view position 
of observers other than themselves, a sight line direction, and the hand part position sensing 
device receiver 103, a direction, and gesture are notified. About a virtual space object, the 
position direction and state of a virtual space object which the virtual space status management 
department 401 manages are notified. Reception of state information is possible always and does 
not need to take timing in particular into consideration. 

[0089]The virtual video generation part 31 1 generates the transparent virtual space image of a 
background seen from the view position of the observer concerned notified from the position 
direction, the state, and the sight line direction amendment software 350 of the virtual space 
object notified from MR space status management software 400, and the sight line direction. 
[0090]The real image acquisition part 312 incorporates a real space image from the object for 
right eyes, and the video capture board 150,151 for left eyes, stores it in the predetermined 
region of memory 157 or HDD155 (shown in drawin g 7) f and updating it is continued. 
[0091]The image synthesis section 314 reads the real space image which the above-mentioned 
portion generated from the memory 157 or HDD155, is overlapped on a virtual space image, and 
is outputted to the display 201,202 of the observer concerned. 

[0092]Each observer can be provided with MR space image with the hardware and software 
which were explained above, 

[0093]As for MR space image which each observer looks at, since the state is managed in a 
unified manner with MR space status management software 400, it is possible to make it 
synchronize in time. 

[0094]Next, it explains, referring to dra wing 9 thru/or d rawing 1 1 for the details of the operation 
of MR image generation software for reducing the time lag of a real space image and a virtual 
space image. 

[009 5] Hardware and software relevant to generation of MR space image of the image generation 
system of drawing 1 in drawing 9, The figure showing the flow of the information typically, the 
time chart drawing 10 indicates the operation timing of MR image generation software of drawin g 
9_to be, and drawin g 1 1 are flow charts which show operation of MR image generation software 
of drawing 9 . 

[0096]If MR space state is notified to the state transmission and reception section 313 from MR 
space status management software 400 in MR image generation software 310 as shown in 
drawin g 9 and drawing 1 1 (Step S100), Instructions of real space image drawing are published to 
the image synthesis section 314, and instructions of virtual space image generation are published 
to the virtual video generation part 312, respectively (A1, A10 which are shown in dr awing 1 0 ). 
[0097]The image synthesis section 314 copies the newest real image data from the real image 
acquisition part 312 in response to instructions (A2 of draw ing 10), and starts drawing on the 
memory 157 of the computer 107 (Step S102). 

[0098]It starts that the virtual video generation part 31 1 creates the position of the virtual body 
in virtual space, a direction, a state, a view position, and a sight line direction in the form of the 
state description form called scene graph based on the information which exists in the state 
receive section 313 (A1 1 of drawing 10; Step S104). 

[0099]Although he is trying to process Steps S102 and S104 in order, it may be made to process 
in parallel in this embodiment using multithread art. 

[0100]That rea! space image drawing processing of the image synthesis section 314 and the 
virtual space image generation processing of the virtual video generation part 312 end MR image 
generation software 310 (A12) And waiting (Step SI 06), After real space image drawing 
processing of this image synthesis section 314 and the virtual space image generation 
processing of the virtual video generation part 312 are completed, instructions of virtual space 
image drawing are gone into effect in the image synthesis section 314. 

[0101]The image synthesis section 314 investigates whether an observers view position and the 
sight line direction are updated (Step S107), Here, when renewal of the above-mentioned 
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information is performed, the newest view position and a sight line direction are acquired (Step 
S108), and the virtual space image which is in sight from this newest view position and a sight 
line direction is drawn (Step S110). The processing which changes this view position and a sight 
line direction, and draws is so small that it can be disregarded compared with the whole drawing 
time, and does not become a problem. When the above-mentioned information is not updated, 
the above-mentioned step SI 08 and S110 are skipped, and drawing of a virtual space image is 
continued as it is. 

[0102]And the image synthesis section 314 compounds the real space image and virtual space 
image which drew, and outputs them to the display 10. (Step S112) 

A series of MR space image generation processings are completed above. Here, this processing 
will be ended, if it investigated whether the end instruction was received and popularity is won. 
The above processing is repeated if popularity is not won (Step S1 1 4). 
[0103]In the midst of a series of above-mentioned processings, the state transmission and 
reception section 314 may receive a new state from MR space status management software 400. 
in this case, it is shown in drawing 10 — as — A1\ A10', and A1 — although that is notified like 
r \ A10", these are disregarded until the image synthesis section 314 investigates whether there 
was any notice at the above-mentioned step S100. 

[0104]Thus, since the state of MR space image which each observer looks at is managed in a 
unified manner with MR space status management software 400, a real space image and a virtual 
space image can be synchronized in time. When two or more observers use this system 
simultaneously by this, it enables each observer to see the image of the same time. It becomes 
possible to use an observer's newest position direction information, and time gap of a real space 
image and a virtual space image can be reduced. As a display which provides an observer with 
MR space image especially, when HMD101 is used, a response when an observer shakes a head 
at right and left improves. 

[0105]As mentioned above, in this system, the time lag of a real space image and a virtual space 
image can be reduced, and an observer can be provided with MR space image which has a feeling 
of devotion more. 

[0106]The storage which recorded the program code of the software which realizes the function 
(the flow chart shown in drawin g 1 1 is included) of an above-mentioned embodiment, It cannot 
be overemphasized that it is attained, also when a system or a device is supplied and the 
computer (or CPU and MPU) of the system or a device reads and executes the program code 
stored in the storage. 

[0107]In this case, the function of an embodiment which the program code itself read from the 
storage mentioned above will be realized, and the storage which memorized that program code 
will constitute this invention. 

[0108]As a storage for supplying a program code, a floppy disk, a hard disk, an optical disc, a 
magneto-optical disc, CD-ROM, CD-R, magnetic tape, a nonvolatile memory card, ROM, DVD- 
ROM, etc. can be used, for example. 

[0109]By executing the program code which the computer read, Based on directions of the 
program code the function of an embodiment mentioned above is not only realized, but, It cannot 
be overemphasized that it is contained also when the function of an embodiment which 
performed a part or all of processing that OS (operating system) etc. which are working on a 
computer are actual, and was mentioned above by the processing is realized. 
[01 10]After the program code read from the storage was written in the memory with which the 
function expansion unit connected to the expansion board inserted in the computer or the 
computer is equipped, It cannot be overemphasized that it is contained also when the function of 
an embodiment which performed a part or all of processing that CPU etc. with which the 
expansion board and function expansion unit are equipped are actual, based on directions of the 
program code, and was mentioned above by the processing is realized. 
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* NOTICES * 

JPO and INPIT are not responsible for any 
damages caused by the use of this translation. 

1 This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2.**** shows the word which can not be translated. 
3.1n the drawings, any words are not translated. 



DESCRIPTION OF DRAWINGS 

[Brief Description of the Drawings] 

[Drawing 1] It is an appearance lineblock diagram of the image generation system concerning one 
gestalt of operation of this invention, 

[Draw ing 2] It is a perspective view showing the composition of HMD with which the head of the 
observer of drawing 1 is equipped. 

[Dr awing 3] It is a figure showing the example of MR space image generation in case all the virtual 
space objects are in this side. 

[Dr awing 4] It is a figure showing the example of MR space image generation when not using a 
penetration virtual space object. 

[Drawing 5] It is a figure showing the example of MR space image generation in the case of using 
a penetration virtual space object. 

[Drawing 6 ] It is a figure showing the example of the gap amendment by the marker in the image 
generation system of drawing 1. 

[Drawing 7] It is a hardware-constitutions figure of the computer 107 in the image generation 
system of drawing 1 . 

[Pr3wi.Qg_.8]Jt is the software configuration figure carried in the computer of the image generation 
system of drawin g 1 . 

[Dr awing 9] It is a figure showing typically the hardware and software relevant to generation of 
MR space image of the image generation system of drawing 1 , and the flow of the information. 
[Drawing 10] It is a time chart which shows the operation timing of MR image generation software 
of drawi ng 9. 

[Drawing 1 1 ]lt is a flow chart which shows operation of MR image generation software of drawing 
9. 

[Description of Notations] 
100a, 100b T and 100c Observer 

101 HMD 

1 02 Head position direction sensor receiver 

103 Hand part position direction sensor receiver 

104 Position direction sensor transmitter 

105 Loudspeaker 

106 Position direction sensor body 
107,108 Computer 

1 1 0 Real space object 
120 Marker 
130 Network 

150,151 Video capture board 
152,153 Graphics board 

154 Serial interface 

155 HDD 

156 CPU 

157 Memory 
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1 58 Sound board 

159 Network interface 

201 A display for right eyes 

202 A display for left eyes 

203 The video camera for right eyes 

204 The video camera for left eyes 

310 MR space image generation software 

31 1 Virtual video generation part 

312 Real image acquisition part 

313 State transmission and reception section 

320 Position direction measurement software 

321 Gesture primary detecting element 

330 Position amendment marker detection software 

340 Sound effect output software 

350 Sight line direction amendment software 

400 MR space status management software 

401 Virtual space status management department 
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1. This document has been translated by computer. So the translation may not reflect the original 
precisely. 

2. **** shows the word which can not be translated. 
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[ Drawing 3] 
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MRSMtttt 
lV7h 



401 



400 



-150 



-151 



312 



311 

.1 



310 



314 



I 



«MB3S5HiW 



^313 



350 
i 



152 
A. 



r 

153 



http://www4.ipdl Jnpit.gojp/cgi-bin/tran_web_cgi_eije?atw^u=http%3A%2F%2Fwww4j.„ 2008/03/25 



JP,2002-157607,A [DRAWINGS] 



5/5 ^— v 



[Drawing 1 0] 
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[Drawing 1 1 ] 
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